Data Integration in Vector (Vertically Partitioned) Databases

Data integration requires not only the absorption of data, but the transformation of that data. For instance, manufacturing companies must combine customer order information for ordering supplies as well as for financial planning; large banks may need to produce consolidated profit-and-loss statements daily, or even more frequently, to manage liquidity and risk. The data may be spread across many heterogeneous databases located in different countries, with different standards of cleanliness, described in different currencies or different units of measure. The transformation rules may also change over time when, for instance, companies merge or split, or new rules of accounting are imposed. Clearly it is best if the data integration tool is flexible enough not only to accomodate new sources of data, but also to change the rules of combining that data. This paper describes one data integration tool, Aleri Inc.’s Modeler. We focus primarily on one aspect of Modeler: the use of vectorization both as an implementation technique and as the fundamental unit of computation. Vectorization improves both the efficiency and the ease-of-use of Modeler.